Click any tag below to further narrow down your results
Links
This page describes the Artificial Analysis Decision Index, a weighted benchmark that scores small "decision models" on fast classification tasks like routing support tickets and rating urgency. It combines six evaluations, from decision-making and tool calling to hallucination and answer invariance, into one composite score.
- Decision models answer every question in a single call with a probability for each option, which keeps them cheap and fast and makes calibration measurable.
- The index weights AA-Paperclip (document extraction as one decision) at 30%, with five other evaluations at 10–15% each.
- Accuracy counts the top-probability option as the answer, and the page also reports Brier scores, time per task, and cost per 1M tasks.
OpenAI has made its Decisions API generally available, a fast, cheap model for picking among options with a confidence score, meant to compete with TypeSafe's Jev. The pitch is that for many workflows, a quick classifier-style model can replace a full LLM.
- OpenAI didn't build a dedicated decision model; it repurposed its ChatGPT Luna model and tuned it for lower latency and cost.
- Speed is the main selling point: response times dropped from 1-3 seconds to roughly 100-180 milliseconds, and Speak cut its action gap from 1.4 seconds to 0.13 seconds.
- Benchmarks don't settle which decision model is best. Jev ranks well on some indices and tops the PacMan-based JevMan test, while OpenAI's model ranks lower, though it may have an edge in image classification.